NTCIREVAL: A Generic Toolkit for Information Access Evaluation

Over the past decades, Information Access (IA) tasks have evolved and diversified. For example, in the mid20th century, Information Retrieval (IR) was about set retrieval for libraries; then with the advent of the digital information overload era, ranked retrieval became a necessity; now in the 21st century, we are experiencing richer forms of IR such as diversified Web search in order to satisfy ambiguous and underspecified queries [25]. Moreover, with the progress in natural language processing, automatic Question Answering (QA) [10, 15, 20] and leveraging Community QA (CQA) data [24] have become feasible. Many of these IA tasks involve automatic ranking of items (e.g. documents or answer strings). To ensure progress in IA research, reliable evaluation metrics are an absolute necessity. Given an IA task definition, an evaluation metric should be designed so that it can guide the system towards the right goal of that particular task. Hence, together with IA tasks, IA evaluation methods and metrics have also evolved and diversified. This paper introduces a tookit for evaluating a variety of IA tasks, called NTCIREVAL, designed primarily for tasks that involve ranking of items. NTCIREVAL is available at http://research.nii.ac. jp/ntcir/tools/ntcireval-en.html. (This paper discusses the version released in April 2011.) It works on UNIX/Linux platforms. While NTCIREVAL can handle some of the ongoing and past IA tasks of NTCIR, the sesquiannual IA evaluation workshop run by National Institute of Informatics, it is a generic toolkit that can be used for other IA tasks. The main objective of this paper is to provide an overview of the philosophy behind and functionalities of NTCIREVAL, so that IA researchers can quickly understand and utilise it whenever appropriate. Because IA research relies much on experimentation, sharing such an evaluation toolkit among the IA researchers should help enhance the reproducibility of experiments, and also foster discussions on how to better evaluate IA tasks. This paper should also serve as a noncomprehensive survey of recent developments in the field of IA evaluation metrics. The remainder of this paper is organised as follows. Section 2 discusses the design philosophy of NTCIREVAL. Section 3 explains how NTCIREVAL can be used for traditional ranked retrieval evaluation and its extensions. Section 4 explains how it can be used for diversified search evaluation. Finally, Section 5 summarises this paper and provides some general recommendations for IA researchers.